User engagement on
NGO Facebook pages

Effects of post sentiment measures over time

Nils Gustafsson & Nils Holmberg

Research questions

  • RQ1: How do Swedish nonprofits’ use of emotional expressions on Facebook change over time?
  • RQ2: What are the effects of emotional expressions on user engagement in Facebook posts from Swedish nonprofits?


  • ca 70 000 posts, after cleaning
  • from 125 pages (e.g. WHO)
  • time period of 2014-2019
  • downloaded using CrowdTangle

Engagement index: sum of likes, comments and shares

Sentiment dictionary

  • content coded for sentiment using the AFINN dictionary
  • better results with machine learning ?
  • quantitative content analysis vs computational

some caption

Sentiment measures

  • ca 70 000 posts from 125 organizational pages
  • time period of 2014-2019
  • text pre-processing, emojis converted to text descriptions
  • sentiment analysis on all text content


  • date of posting
  • number of followers
  • post language

Phrases like “tears of joy” better handled with e.g. VADER package?

Sentiment analysis

  • content coded for sentiment using the AFINN dictionary
  • words and emojis assigned valence value of -5 to +5
  • valence indicate positive or negative sentiment (0 being neutral)
  • valence squared indicates intensity of sentiment

some caption

Sentiment over time (RQ1)

  • sentiment normalized within organizations
  • normalized sentiment aggregated per month
  • positive relationship between sentiment and time?

some caption

Engagement and sentiment (RQ2)

  • engagement index: sum of likes, comments, shares

  • engagement index normalized within organizations

  • normalized engagment aggregated per month (mean)

  • positive relationship between engagment and sentiment?

some caption

Effects on engagement

  • positive effects
    • post sentiment valence
    • post sentiment intensity
    • number of followers at time of posting


  • negative effects
    • time, month number 1-60


  • type of organization?

Linear mixed effects model with organization as random factor

Variables by organization type

  • 1=culture and recreation
  • 2=education and research
  • 3=health
  • 4=social services
  • 5=environment
  • 6=development and housing
  • 7=law, advocacy an politics
  • 8=philantropic intermediaries and voluntarism promotion
  • 9=international

Engagement, mean over time

  • big differences between organizations
  • green: org_type “health”, large org
  • red: org_type “international”, small org

Engagement, trend over time

  • big differences between organizations
  • green: org_type “health”, large org
  • red: org_type “international”, small org

Engagement measures

  • engagement index, dependent variable

    1. e_index_mean: per post e_index average per month (“raw engagement scores”)

    2. e_index_mean_norm: normalized per organization values of e_index_mean (1. above)

    3. e_index_norm_mean: per post e_index normalized per organization average per month

    4. e_index_rel_mean: per post e_index relative to follower count average per month

Engagement trend by organization type

  • linear mixed effects model, random factors org type and time

  • structural equation modelling, latent growth analysis

  • poisson regression, measure distributions

Engagement measures by organization type

  • linear mixed effects model, random factors org type and time

  • structural equation modelling, latent growth analysis

  • poisson regression, measure distributions

Engagement by org type and time

  • engagement measures
  • heatmap visualizations

Thanks!

Presentation

  • Nils Holmberg


  • Strategic Communication

    • computational content analysis

  • Cognitive Science

    • cognitive communication effects

https://nils-holmberg.github.io/scom-org/web/pre-socm-sa.html